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ABSTRACT 

The S--iple Design and Data Collection Procedures: 
National Sttidy of the Upward Bound Program* 
Dr. Jane W. Bergsten 
Research Triangle Institute 

The Research Triangle Institute conducted an evaluation study of the 
Upward Bound Program, a nationwide program funded by the U.S. Office of 
Education to help selected low-income high school students prepare for and 
enter post secondary education. This paper, the third in a series of four, 
describes the sample design and selection procedures used in selecting a 
sample of Upward Bound students and a sample of comparison students for 
the study. A multistage probability sample design was used. In addition, 
data collection procedures, which included questionnaire. administration, 
mail queries and telephone followups, are described. 



*Paper presented at the 1976 Annual Meeting of the American Educational 
Research Association, San Francisco, California, April^ 23, 1976. 



SAMPLE DESIGN AND DATA COLLECTION: 
NATIONAL STUDY OF THE UPWARD BOUND PROGRAM 

A. Sampling Overview 

A major purpose of this study was to measure the effect of the Upward 
Sound (UB) program on the educational continuance rates of the high school stu- 
dents It serves.-^ In order to measure this effect, It was necessary to have 
some standard against which to compare the rates of the Upward Bound 
participants. The study, therefore, also concerned Itself with measuring 
educational continuance among students that were similar enough to the 
Upward Bound participants to provide meaningful benchmarks again which rates 
of Upward Bound participants could be compared* 

Ideally, a researcher wotild set up an experimental design in order 
to measure the effectiveness of such a program as Upward Bound. Students 
wotild be identified as meeting the acceptance standards of the program, 
and those eligible wotild be. randomly^ assigned to the actual program or 
to a control group. Both the program participants and the control group 
students wotxld be observed for a number of years, and the continuance rates 
calctilated. By comparing the continuance rates for the Upward Bound program 
participants to those of the control group, the effect of the program on 
educational continuance could be estimated. Such types of experiments are 
rarely feasible and in many cases are administratively and pollclcally 
impossible. Often, as in this case, a program has to be evaluated after 
it has been in effect for a number of years. However the evaluation pro- 
cedures available at that stage cannot provide the precise answers that 
would be forthcoming from an" experimental design. 

This sttidy design, and the associated sample design, foctised on the 
effect of the Upward Botmd program on educational continuance rates, as 

— '^The three main objectives of the study were to measure the effect 
of the UB program on (a) the high school retention rates of its participants 
(b) the post secondary institution entry rates of its participants and (c) 
the generation of the skills and motovatlon necessary for success and 
education beyond high school. 
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measured during a short span in time, after the program had been in oper- 
ation for a number of years. To obtain measures of program effect the 
study measured the contintiance rates, over the course of a one year period, 
of (1) Upward Bound participants who were high school sophomores, jioniors, 
or seniors at the beginning of the one year period and (2) comparison students 
in these same three grades. 

Ideally, such rates as described above would be computed for the 
entire population of Upward Bound participants in the grades of interest 
and for the entire population of comparison students. However, collecting data 
on an entire population is often both administratively cumbersome and 
financially impossible. Since statistical procedures exist whereby unbiased 
estimates of population characteristics can be obtained from data based 
on only a sample from the poptilation, rather than the entire population, 
a sample survey is often the best way to obtain the desired information. 

A probability sample can be selected in such a way that, not only can 
unbiased estimates, such as number of tenth grade dropouts, be made from 
the sample data, but estimates of sampling errors can be calculated as well. 
Sampling error is error that must be tolerated because we choose to select 
and measure only a sample of elements rather than all elements in a popu- 
lation. The sampling error (or standard error) provides a measure of the 
range within which a sample estimate can be expected to fall a certain 
percentage of the time. The magnitude of the sampling error is related 
.to two things over which the sampler can exert some control, namely, the 
size of the sample and the procedures used in selecting the sample. 

In developing a sample design, the sampler is concerned with the 
selection of a sample that will yield estimates of sufficient precision. 
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i.e., estimates having a small enough sampling error so as to be useftil, 
and with producing these estimates for the least cost. In most instances 
a sample that is widely spread over the entire population will produce the 
most precise estimates. However, the cost of conducting a study with 
such a widespread sample is usually much greater than the cost of a study 
using a cluster sample (i.e., a sample where several or many elements are 
selected from the same place) . By balancing both expected sampling 
errors and expected costs, the most desirable sample design can be deter- 
mined. 

For the Upward Bound evaluation study, rather than selecting a sample 
of Upward Bound participants without regard to their location, the selection 
was made by first selecting a sample of Upward Bound projects . In determin- 
ing the number of projects and the number of Upward Bound students to 
include in the sample, the goal was to produce the most precise estimates 

for the least cost. 

Since the purpose of the study necessitated the estimation of contin- 
uance rates not only for Upward Bound students but also for the comparison 
group of similar students, a comparison poptilation had to be defined and 
-sampled. It was decided that a group that satisfied both the requirement 
of similarity and of comparability to the Upward Bound students was the pop- 
ulation of students attending the same schools as the Upward Bound students . 
Choosing this definition of a comparison group had both analytical and 
administrative advantages. By using as comparison students those who attended 
the same schools as UB students, we were controlling the differential 
institutional effects. This can be expected to have the effect of 
increasing the precision of estimates of differences between UB students 
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and comparison students* In addition, using such schools provided for geo- 
graphic clustering of the sample UB and the sample comparison students and an 
associated reduction In cost* 

The proposed definition of comparison students has one obvious disadvantage. 
To the extent that UB activity in a school has had a beneficial effect on 
students who have in no way been connected with the UB program, the study 
results would be biased in the direction of underestimating the beneficial 
effect of the UB program on educational continuance rates. However, all 
things considered, the proposed definition was judged to be the most desirable 
and the comparison group of students was therefore defined as 10th, 11th, and 
12th grade students who were not in a UB program but who were attending 

schools that were also attended by Upward Bound students. These schools will 

2/ 

be referred to as "feeder" schools.— 

Considering the precision of the estimates to be made from the samples, 
the costs Involved in conducting the study and the administrative feasibility 
of carrying out field procedures, it was decided that 54 of the 333 Upward 
Bound projects would be selected into the sample. All students who were 
Upward Bound participants in a sample project during September of October 
.1973 were Included in the sample. In addition, for each sain)le project, 
two "feeder" schools were selected and from each "feeder" school six or 
seven comparison students were taken from each of grades 10, 11, and 12. 
To further control the amount of field work Involved in the sampling pro- 
cedures, it Was decided to sample students within the selected schools in 
two stages, by first selecting a sample of classes and then selecting a 
sample of students within the selected classes. 

— ^For the schools actually selected into the sample, an average of 
only 1% of the 10th, 11th, and 12th graders were actually Involved with 
a UB program. The spillover effect of the UB program on the non-UB 
students in the same school might actually have been minimal. 



■ 

y In addition to the student samples, two different sample of staff 

: members were selected: (1) staff from Upward Bound projects, and (2) staff 
: from "feeder" schools. In addition, a subsample from the sample of 54 

projects was selected for site visitation • The procedures for the selection 
:;. of the UB and comparison student samples will be described in some detail. 

The other sampling procedures will not be covered in this paper. 

A graphical description showing the different stages of selection of 
. the UB and comparison student samples is presented in Figure 1. The pro- 
: cedures used at each stage of selection are described In the following sections. 
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B, Sampling Proctdurti— 

!• The UB Prolact Sampla 

Research Triangle Institute (RTI) obtained, from the National Office 
of Education (OE) and from the ten regional OE offices, copies of the project 
proposals for all UB projects that were funded for program year 19 73-74 L 

From these proposals characteristics of the funded projects were obtained, and 
a frame from which projects would be sampled was constructed • To have 
been included in the project sampling frame a project must have possessed 
all of the following characteristics: 

a) The project was funded for fiscal year 1973, 

b) The project proposed to serve students in grades 10, 11, 
and/or 12. -^^ 

c) The project was located in coterminous United States. 

The 333 projects possessing all of the above characteristics constituted 
the frame from which sample projects were selected. Using project charact- 
eristics obtained from the project proposals, the 333 projects were 
partitioned Into 27 strata on the basis of six variables. These variables 
together with their code categories, are listed below: 

a) Ethnic background of participants. 

(1) Black. 

(2) Black and white. 

(3) 10%+ American Indian. 

(4) 10%+ Mexican American. 

(5) 10%+ Puerto Rican. 

(6) All other. 

— ^Por more detailed discxission of the sampling techniques described 
in this report, see Klsh, L. (1965), Survey Sampling . New York: John Wiley 
& Sons . 
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Offlct of Education gtographlc rtglon 
(Raglon No. 1 through No. 10) 
Program emphasis. 

(1) Academic orientation. 

(2) Vocational orientation* 

(3) Combination of academic and vocational orientation. 

(4) Other. 
Project age. 

(1) New this year. 

(2) One year old. 

(3) Two years old. 

(4) Three or more years old. 
Location of participants. 

(1) Cltywlde 

(2) Selected parts of a city. 

(3) "Rural. 

(4) City and rural. 

(5) Statewide. 

(6) Reservation. 

(7) Regional. 

Type of sponsoring Institution or agency. 

(1) Secondary school. 

(2) Two-year college. 

(3) Four-year college. ----- 

(4) Vocational-technical school. 

(5) Private, nonprofit agency. 
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(6) Consortium of •ducatlon Initltutlona • 

(7) Agtncy for a consortium. 

(8) Proprlttar/ school. 

(9) Public agency. 

Tha strata ware constructed so as to have approximately equal total 

number of participants In grades 10, 11, and 12. From each of the 27 

approximately equal-size strata, two different projects were selected for 

the sample with probability proportional to a measure of size which . 

4/ 

reflected the total number of participants In grades 10, 11, and 12— • 
This resulted In the selection of 54 different UB projects. 

It may be helpful to consider the stratification and selection of the 
54 sample projects In more detail. Stratification may be defined as the 
dividing of a population Into sub-parts called strata, for the purpose of 
sampling separately from each stratum. Although there are a number of 
reasons for stratifying prior to the selection of the sample, the 27 strata 
were formed for the selection of 54 sample Upward Bound projects for two 
basic reasons: (1) to Insure that certain minority groups such as American 
Indian, Mexican-American, and Puerto Rican would be properly represented so 
that there would be a sufficient number of such students in the sample to 
enable us to make relatively reliable estimates about these groups from the 
sample data; and (2) to attempt to reduce the size of sampling errors 
that otherwise would result if a sample of 54 projects were selected without 
stratification. 

In forming the 27 strata, the aim was to make the projects within each 
stratum as similar as possible on the variables of central interest in the 
study, e.g., education continuance rate for tenth grade UB participants,' 

-^^The procedure used is described in detail in M. R. Samford, "On 
Sampling Without Replacement with Unequal Probabilities of Selection," 
Biometrika. 54, 3 and 4 (1967), 499-512. 
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college entrance rate among twelfth grade UB participants, etc., and 
to make the 27 strata as different from one another as possible, on 
these variables. To the extent that this is done successfully, the 
sampling errors will be decreased, as compared to those based on an 
unstratified design. 

Ideally one would form strata on the basis of the central variables 
themselves, but because such information is in general not available, the 
strata are usually formed on the basis of characteristics thought to be 
highly correlated with the central variables of the study. To the extent 
that stratifying on these related characteristics actually does form strata 
that are, in fact, homogenous with respect to the central variables under 
study, the sampling errors of the estimates of these variables will be 
reduced. 

The stratification variables used in dividing the 333 projects into 
27 strata were those thought to be related to school continuance 
rates: ethnic background, geographic region, Upward Bound project program- 
emphasis, project age, location of participants, and project sponsor. If 
all of these six variables were cross classified with one another, -there- - 
would be some 108,000 cells. Since there are only 333 projects, most of 
the cells would be empty, and many would contain only one or two projects. 
It is obvious that considerable combining of cells was necessary. The 
combining process resulted in certain categorizations being used for some 
groups and not for others. For example ^ city versus : jn-city was a meaningful 
and useful way to divide projects having a sizeable number of Puerto Ricans, 
but was not useful in dividing projects classified as American Indian. For 
this latter group, a reservation versus non-reservation distinction was made. 
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la forming the homogeneous strata, an attempt was made to form the 
iii/ strata so that they were as nearly equal In size as possible (i.e., 
i^; equal with respect to the number of Upward Bound participants antici- 

pated for the school year 1973-74). This was done because, in general, 
A using eqxial size strata can be expected to bring about greater gains in 
precision of estimates relating to student characteristics than using 
strata that very greatly in size. 

Twenty-seven strata were used for the selection of the 54 sample 
projects, with two projects being selected from each stratum. This set 
of 27 strata, of course, form only one set from among the thousands of 
possible sets of 27 strata that could have been formed. It is entirely 
possible that another set would have been better, that is, would have 
yielded estimates having a somewhat smaller sampling error than estimates 
from a sample based on the 27 strata that were actually used. Undoubtedly . 
many other sets would have been worse, that is, would have provided estimates 
with somewhat larger sampling errors. Different ways of forming 17 strata 
might affect the size of the sampling errors of the estimates, but in no 
~ way would the unbiasedness of the estimates that can be made from the study 
data be affected. Unbiased estimates of population characteristics are 
possible regardless of the effectiveness of the stratification. 

Once the 27 strata were formed, 54 projects were randomly selected, 
two from each stratum using probabilities proportional to a measure of 
/size and without replacement. By selecting two projects from each stratum 
it is possible to make unbiased estimates of error variances. 
2. The Student Samples 

a. The UB Student Sample 

Within each of the 54 saoqple projects, all students who were in 



UB participants in September or October of 1973 were selected into the 
sample. This resulted in the selection of the 3710 eligible UB students 
associated with the 54 sample UB projects* 
b * The Feeder School Sample 

A sample of control or comparison students against which to compare 
the UB students was selected. The comparison students as well as certain 
staff members were selected from "feeder" schools associated with the 
54 sample projects. A feeder school was defined as a school with at 
least four 10th, 11th, and/or 12th grade students listed on the most 
recent CUB^'' roster as being clients of a given UB project, or as a 
school from which a newly funded UB project planned to serve at least 
fotxr 10th, 11th, or 12th grade studepts during the 1973-74 school year. 

For sampling purposes each "feeder" school was associated with one 
and only one UB project. If a "feeder" school "fed" more than one 
project, it was associated only with that project to which it provided 
the greatest number of students. In order to accomplish this procedure 
of uniquely associating a "feeder" school with one and only one UB 
project, the CUB roster which listed the schools attended by UB students 
was checked for all 333 projects in the defined population. 

Once the "feeder" schools for a sample project had been identified, 
they were listed in descending order according to the number of UB students 
who attended the school and who were associated with the sample Upward 
Bound project. The estimated total number of students in grades *10 through 
12 was then recorded for each "feeder" school, and these numbers were then 
accumulated. A systematic random selection of four "feeder" schools was 

-5^The Current Upward Bound Student Roster that was correct as of 
^August 1973. 



then made with probability proportional to size. An equal probability 
sub-selection of two of the four schools was then made and designated as 
"sample" schools. The remaining two schools were designated as "backup" 
schools to be used only if the "sample" schools ref\ised to participate in 
the study. For som^ projects additional "feeder" schools were selected. 
This was done to keep the comparison student weights from being excessively 
large. 

A total of 113 school selections were made, bringing into the sample 
108 different schools. The selection procedures permitted a school to be 
selected more than once, and in five instances a school was selected twice. 
In such cases, the number of sample students to be selected from the school 
was doubled. 

Of the 108 sample schools selected, five refused to participate. Four 
'^backup" schools were selected as replacements, which brought the total 
to 107 schools that were expected to participate. The late refusal of one 
school resulted in a sample of 106 schools providing the sample comparison 
students that were used in the study. (See Table 1) 

c , " The Comparison Student Sample 

Within each sample "feeder" school, six homeroom classes of 
students in grades 10 through 12 were selected with equal probability from 
class lists supplied by the sample schools. For each selected class a 
list of students who were in the class during September 1973 was obtained. 
For each student listed, the homeroom teacher was asked to indicate the 
student's grade level and ethnic background, and to make an "educated 
guess" as to whether or not the student came from a low income family and 
as to whether or not the student should be considered as an "academic risk" 
for a two- or four-year college education. This information was used to 
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1^: Table 1 

Distribution of sample feeder schools by study participation 





Nimiber of schools 
in sample 


Number of school 
selections la 
sample* 


Nuniber selected into sample 


108 


113* 


Number refusing to participate m scuay 




5 


Nuniber of back up schools selected as 
replacements for reftxsals. 


4 


5** 


Total in adjusted sample 


107 


113 


Number refusing to participate in 
study (during data collection stage) 


1 


1 


Total number participating in study 


106 


112 



: *The sampling procedures permitted a school to be selected twice, in which case 
: the number of students to be selected from the school was doubled. The 113 
^ school selections resulted in the selection of 108 different schools, 5 of 
;^ which had been selected twice. 

L**To adjust for the 5 schools that had refused, 4 "back up"schools were selected 
r to replace 4 of the 5 refusals. In the case of the fifth refusal, the sample 

size In the cooperating sample school associated with the sample project was 
? doubled. 
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stratlfy students prior to selection. (Rather than sending class lists, 
several sample schools sent complete rosters of all of their students 
who had been enrolled in September 1973. In such cases a sample of 20 
students from each grade level was selected, and the lists of selected 
students were then sent back to the school with the request that the 
school provide information on the background characteristics of each student 
listed. Such completed lists were then treated as though they were class 
lists, for the purposes of selecting the sample comparison students for 
the study.) 

The student lists for the sample classes were carefully checked against 
the September 1973 list of UB students, and the students who were listed 
as being associated with an Upward Bound project were removed from the 
student class lists. From these corrected lists, a stratified random sample 
of comparison students was then selected, with an expected 21 students 
from each sample control school, yielding a total of 2340 eligible comparison 
students. 

The selection of the approximately 21 students per school involved 

a predetermined set of procedures. First, using the information«,proyided 

by the homeroom teacher, the students were stratified according to grade- 
in-school and according to whether or not they appeared to meet the 
eligibility requirements of the UB program. Then;.*:* random sample of 
students was selected from each stratum, with the objective of selecting 
approximately eqxial numbers of students from each of the three grades, and 
of over-sampling students who appeared to be like UB students. While 
we thus selected into the sample students tentatively classified as "like UB" 
as well as those tentatively classified as "unlike UB" students, we included 
in our sample a disproportionately high number of "like-UB" students. 
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This was done in order to attempt to insure the inclusion of a substantial 
number of students who were, in fact, eligible for an UB program, (In the 
analysis of the study data, the characteristics of comparison students were, 
of course, determined from different factual information that the teachers 
"educated guesses" that were used in the sample selection procedures •) 

• The approximate relative sampling rates for students in each of four 
income academic risk grotips is presented in Table 2. 

Table 2 

Approximate Rates of Selecting Comparison Students 



Family Income 

Not Classified Low Family 
as "Low" Income 



Academic risk 1 x rate 4 x rate 

Not Academic risk 1 x rate 2 x rate 

Note that "low income-high academic risk" students were selected at approximately 
four times the base rate and other low income students were selected at 
Approximately twice the base rate« 

At each stage of the sample selection, each, unit in the defined popul- 
ation was given .a positive chance of being selected. These probabilities 
were recorded and combined into an overall probability of selection for each 
UB and each comparison student, thiis providing for the computation of 
unbiased estimates of population characteristics for UB students as a whole 
as well as for comparison students as a whole. In order to produce unbiased 
estimates, each student in each of the two samples was ultimately assigned 
a weight equal to the inverse of the probability of his being selected. Thus, 
students selected with smaller probabilities received larger weights, and 
vice versa. The weighting procedures used at the estimation stage 



compensated for unequal probabilities having been used at the selection 
stage, and permlted the unbiased estimation of characteristics of the 
poptilatlon of UB students and of the population of comparison students. 
In addition to making unbiased population estimates the study plan called 
for making comparisons of the two groups of students, 'the U? and comparison 
students. This latter population, however, was not necessarily expected to 
be similar to UB students on all revelant background characteristics. In 
order to reduce or eliminate the effects of these dissimilarities, a balancing 
or standardizing statistical technique was planned and ultimately employed. 
This technique, which will be described in some detail, statistically 
adjusted the comparison student population to the UB population using 
techniques similar to those employed by demographers when they construct 
"adjusted", "standardized", or "corrected" birth rates or death rates. 
The aspect of the sample design related to the oversampling of comparison 
students thought to be "like" UB students aimed at Increasing the precision 
of the comparisons between UB students and the "balanced" or "standardized" 
comparison student population. The balancing procedures and estimation 
techniques will be described in more detail in a later section. :^ 
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; C. Data Collection Procedures 

Although the populations to be sanxpled were defined according to 
their UB participation and school attendance status as of September or 
October, 1973, the actual data collection did not begin until the spring 
of 1974, at which time the sample UB and sample comparison students 
were queried. 

Prior to the spring data collection Initial contacts had been made 
with the UB project directors and with state and local school officials 
In areas where sample schools were located In order to secure their 
endorsement. Sample school personnel were then contacted to obtain Infor- 
mation for sampling purposes and to pave the way for the data collection 
procedures. During the Initial contacts with a sample school a "school 
contact" Individual was appointed to serve as a laison between RTI and the 
school. The school class lists, the student lists for the sample classes, 
etc. were obtained through the "school contact". 

Sixty-two study administrators were hired to conduct questionnaire 
administrations and to collect school transcript data at the 54 UB projects 
and 107 sample feeder schools. Six regional training sessions were, held 
to provide the study administrators with the study objectives and to 
familiarize them with the forms and procedures to be used in carrying out 
the questionnaire administrations, obtaining transcript information, etc. 

The Basic Student Questionnaires (BSQ) were administered to UB students 
(l) in group sessions during regtilarly scheduled meetings of ftill UB membership 
at a sample project and (2) in remote site administrations where projects 
held only Infrequent meetings or where the meetings were poorly attended. 
In addition, the study administrators collected basic transcript infor^ 



from the project files. The BSQ were administered to comparison students 
In group sessions held at the sample feeder schools, and transcript Infor- 
mation was obtained for sample comparison students from school files by the 
study administrator. For both UB and comparison students, make-up sessions 
were held for those students who did not show up for the initial questionnaire 
administration sessions. 

Drop-out-Transfer Questionnaires were mailed to "absentee" sample UB 
and comparison students, and an additional mail follow-up was sent to non- 
respondents . 

There was great concern on the part of RTI staff members to Insure a 
high response rate, and steps such as scheduling .extra make up sessions 
during UB summer programs and mailing BSQ questionnaires were taken. 

The difficulty encoimtered in obtaining a high response rate during 
the spring data collection caused some .concern about the prospects for the fall 
status questionnaire that would be mailed out during the early fall of 
1974. A small pilot study was conducted during early Septemb.er 1974 to 
determine the feasibility of using a telephone interview to obtain the 
desired s The results of pilot study convinced us that 

the FSQ response rates would undoubtedly be low, and that they wotild be 
especially low among previous non-respondents. . ^ 

The mail returns of the fall status questionnaire did Indeed fall far 
below what had originally been expected. The low response rate on the 
FSQ caused more than the usual amount of concern becaxise it was felt that 
whether or not the sample UB or comparison student responded might well be 
highly related to whether or' not the person was still enrolled in school. 
Such a relationship cotild bring about spurious results if estlMtes Wre based 
on approximately 50 percent responding. 



It was reasoned that a UB or comparison student who had dropped out 
T of school wotild be more likely to have left home and less likely to have 
received and responded to the mall FSQ. If such were the case, the school 
continuance rate among non-respondents cotild be far less than the con- 
tinuance rate among respondents to the FSQ. 

Special steps were taken to obtain fall status Information from those 
UB and comparison students not , Responding to the FSQ. A telephone tracing 
procedure was employed wherein the sample UB or comparison student was 
telephoned and the desired Information was obtained for the sample Individual 
by means of a telephone Interview. In cases where the individual himself 
could not be contacted, the desired Information was sought from other 
knowledgable Individuals such as parents, project directors, school personnel, 
etc. 

Although It would have been preferable to contact all non-respon- 
dents to the FSQ by phone, cost considerations made this prohibitive. 
Instead, a sampling plan was Instituted. The most worrysome groups were 
those UB and comparison students from whom no response to any questionnaire 
^ad yet bniM'" obtained . ni'^sucH^stude^ 

: refusals) were subselected with certainty Into the telephone tracing 

n subsample. For those students, not only was the FSQ Information collected 

by phone, but also certain pertlnant Information that shotild have been 
^ collected during the previous spring. Of the remaining non-respondents to 
J the FSQ, all of whom had responded to a previous query, approximately 40 per- 
iiXcrat were randomly selected for Inclusion In the telephone tracing stAsioq^ 
r Oompenaation for the subselectlon procedure would be. made prior to estimation 

by a special weighting 'procedure. 

T^ sizes and responses received are detailed In table 3. ^ 



Table 3 



Instrument Response Rates 



Group 



Instrument 



Number 
Eligible 



Percent return 
for all 
eligibles 



Upward Bound 



SPRING 1974 DATA COLLECTION 

Basic student questionnaire 
Dropout transfer questionnaire 

Student transcript form 



3,337 
373 

3,710 
3,710 



82. 8; 
37.8!5 

100.0% 



Comparison 



Basic student questionnaire 
Dropout transfer questionnaire 

Student transcript form 



2,082 
258 

2,340 
2,340 



85. IZ 
25.6% 

99.1% 



FALL 1974 DATA COLLECTION 



Upward Bound 



Fall status questionnaire 
Mail: previous respondents 

previous nonrespondents 



Telephone : previous 

respondents 



previous non- 
respondents 



3,179 
531 

3,710 

1,183 

414 



62.8% 
22.0% 



38.6%* 
97.8% 



Comparison 



Fall status questionnaire 
Mail: previous respondents 

previous nonrespondents 



Telephone : 



previous 
respondents 

previous non- 
respondents 



1,838 
502 

2,340 

767 
424 



58.3% 
15.5% 



38.3%* 
90.6% 



*Reflects primarily subsantpllng. Response rate among those sampled was 
approximately 95%. 



D. Estimation 



Since all of the samples selected for this study are probability 



|i; samples. It Is possible to make unbiased estimates of population character- 
1^ Is tics from properly weighted sample data. The weights used In making 

i these estimates are deteimined from the probabilities of selection. Each 

i 

element In the sample was assigned a weight eqiial to the Inverse of the 
probability of Its being; selected in the sample. 

For example, to obtain weights for comparison students, we first deter- 
mined the overall probability of selection for each comparison student. 
These were determined as follows: 



irall pro- 
ilUty of 
jection of 
qparlson 
uient 



Probability of 
selecting the 
UB project 
associated 
with his 
, school 



Probability of 
selecting his 
school, given 
that the 
associated 
project had 
been selected 



Probability of 
selecting his 
class, given 
that his 
school had 
been selected 



Probability of" 
selecting the 
student, given; 
that his class; 
had been 
.selected 



1^ The weight assigned to a student was then determined by taking the inverse 
r~^of his overall probability of being selected. ... — _ 



Si 



(Student weight) 



(overall probability of 
selecting the student) 



K Each DB participant, comparison, student, (and. staff membier) selected 

t-^^ ■ ■■- • ■ . v/f-^'-ir^- ■ ■ 

Into r.he saiiq)le was assigned a weight, which was. computed from his overall 

0:^. ! ■' ■ • ; ■ ""'■■■■"■■■-■■■-■^■y;:-^:';V- ■ ; . 

p!^ probal?lllty of selection. (In addition, each of the 54 projects was 

i assigned a weight, and each of the 15 projects subselected for site visitation 

i^:- was assigned a second weight .) All population estimates^ maderfrom-the^sample - 



iy- data used these sample weights.— 

I;- 

If responses were available for every sample individual in a specified 
^ subgroup, for example U3 students, an estimated total could be obtained 
f by merely summing the weights of the sample individuals in that subgroup. 
I Because there was some non-response, a special weight 

adjustment was applied to responding sample students, to compensate 
the nonresponses . Non-responses adjustments were made within homogeneous 
• categories, such as grade-project categories for UB students, and project- 
grade-balancing group;^'' categories for comparison students. 



Detailed formulas for estimating population totals, means, and 
^ proportions and their respective sampling errors are presented in: Graham 

J. Burkheimer, et.al. Evaluation Study of th e Upward Bound Program 
hi (Volume IV of A^tudy of the National Upward Bound and Ta lent Search Programs 
r "Four Volumes) R^earch Triangle Park, North Carolina: Center for Educational 
H. Research and Evaluation, Research. Triangle Institute, March 1976 . 

i'T- '. . '• ^ '.*.',-' " ; 

l!' -The balancing groups, which were formed on the basis of ethnicity, sex 

poverty status, ^and academic risk status, are defined more completely: i 
fev later- »«ctiott. 



fJTy *; . . • ....... 

|:E. Standardization or Balancing 

The population of comparison students. I.e., the group against which 

the UB students were to be compared, was defined as all non-UB students 

t- 8/ 
enrolled in the 10th, 11th or 12th grade of an UB project feeder school- 
in September 1973. There was no reason to believe that the distribution 

^of ~the- population of comparison students on relevant background..characteris.tlcs _ 

would be exactly the same as those of the UB population of students. Therefore, 
any direct comparisons between UB students and comparison students with 
respect to such variables as school continuance rates would not take account 
of the fact that the distributions of the groups being compared might be 
different on characteristics such as family Income, ethnic background, etc. 
An adjustment for such differences in these background characteristics was 
made by using procedures similar to those used by demographers in constructing 
what are referred to as "adjusted," "standardized," or "corrected" birth 
rates or death rates.— ^ The demographer, for example, computes age specific 
birth rates for each relevant age group, and applies them to a^popixlatlon of 
Interest, age group by age group, in order to obtain an "age adjtisted" 
birth rate. In a similar fashion we could, for example, compute school 
continuance rates for each of several family Income categories, for lOth 
grade comparison students. By applying these rates to a population of 10th 
grade comparison students having the same Income distribution as the UB 
student population, an "adjusted continuance" rate could then be computed for 
comparison 10th grade students. The comparison of the continuance rate for 

UB students with the "adjusted" continuance rate for comparison students wotild 

. • '■ ■ 

be free of the effects of differential family Income distributions among UB 

10th graders as compared to comparison 10th graders. By computing such "adjusted" 

rates the effects of differences in background charactenstics of the OT^^^^^ 
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A hypothetical example is presented in Table 4 to show how different 
background characteristics could affect the results of comparing two groups 
such as UB and CS, and how a balancing or standarization procedure could 
be employed. In Table 4 the proportion of students in each of the three 
-racial groups is presented for UB and CS students. In addition, a fictitious 
"average-scale-score" is given for each of the six race by type-of -student 
subgroups. Note that when the average-scale-scores for UB and CS students 
are compared within each of the three racial groups, it is clear that the UB 
students have lower average scores. Consistently, in each of the three racial 
groups, the average scale score for UB students is 10 points lower than for CS 
students. For Blacks the UB to CS comparison is 100 to 110, for whites it is 
70 to 80 and for "other" it is 110 to 120. Note, however, that the average- 
scale-scores for all races combined show a reverse relationship, 97 for UB 
as compared to 95 for CS. The reversal in the relationship is brought about by 

Table 4 

HYPOTHETICAL EXAMPLE OF BALANCING TECHNIQUE 



Race 


Upward Bound 
Proportion Average 
as students scale 
score 


Comparison 
Proportion 
of students 


Average - 
scale 
score 


Balanced Comparisons;^ 
Proportion Average p 
of students scale .- 
score' 


Blacks 


.61 


100 


.28 


110 


.61 


no . J 


White 


.18 


70 


.55 


80 


.18 


80 % 


Other 


.21 


110 


.17 


120 


.21 


120 .^^1 


All races 


1.00 


97* 


1.00 


95** 


1.00 


107***:| 



The average-scale-score is obtained as follows: 
( . 61) (100)+( . 18) (70)+( . 21) (110) -97 

^The averase-scale-score is obtained as follows: 



m ' ■ 

9^- ' 26- 

! the differential racial distributions of the UB and CS grotips. The differences 
in racial makeup of the two groups have obscured the tnie situation. A 
balancing technique which applies the average-scale-scores of the CS group to 
a CS population with the same racial distribution as the UB groups yields an 

"'^average scale score of 107 for CS. " This permits a comparison of UB and CS 

average-scale-scores that is free of the effects of differential racial 

distributions. Note that the UB-CS comparison on average-scale-score, 

after balancing, is 97 vs. 107. This reflects the 10 point difference in 

scores that appeared in each of the three racial groups. 

While the actual balancing technique that was used in the analysis 

IftYQlved more than three balancing categories, the example shows the general 

way in which the balancing technique was applied. 

The procedures actually used involved standardizing or balancing the CS 

group of students to the UB population characteristics within each of the three 

grades 10-12. For each of grades 10, 11 or 12 separately, 16 balancing groups 

were formed for UB students, and again for CS students. The 16 balancing 

groups were based on two variables associated with UB participation qualifications, 

namely poverty status and academic risk, and on two additional vai;iables, 

ethnicity and sex. The 16 balancing groups were formed as follows: 

Table 5 

Definition of Balancing Groups 

Poverty Status and Academic Risk 

Poverty Poverty Not 'Toverty" Not 'Toverty" 
Ethnicity and Sex Risk Not "Risk" Risk Not '^sk" 

Balancing Group Number 

Black, male 12 3 4 

Black, not "male" 5 6 7 8 

White, all sexes 9 10 11 12 
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. Within each of the three grades separately, for each of the 16. balancing 
; groups, the total estimated population size was obtained for UB students and 
for comparison students. Using these numbers a balancing weight component 
was computed for each comparison student. Each comparison student's final 
sample weighJA^ was then multiplied by his balancing weight component to 
— obtain his final balanced sample weight. 

The computations were thus carried out as follow.?: " 
For a given grade, g-10, 11 or 12 let 

Y (k) « the estimated number of UB students in grade g in balancing group k. 
ug'' 

Y (k) « the estimated number of CS students in grade g in balancing groiip k. 
eg 

Y (k) 

(]^\ « Jiit- — a proportion of grade g UB population in balancing group k 
ug 16 

S Y (k) 



Y (k) 

(It) « — > proportion cf grade g CS population in balancing group k. 
eg Id 

2 Y^„(k) 
k-1 

The balancing weight component for each comparison student in grade g in 
balancing group k is then 

fl u (k) 

''cg^'^^ a.^g(k) 

By multiplying his final sample weight by W^^W^ the comparisons students 
final balanced weight was obtained. 

Balanced or standardized rates, proportions, etc. could then be estimated 
for the comparison students in a given grade merely by using the final 
balanced weights rather than the final sample weights hen applying the 
estimation formulas. The comparing of UB rates, proportions, etc. with 
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then be free of the effects of differential distributions among the 
balancing groups .—^ 



